Move APRS tests to JSONL case vectors, compared against FAP and live traffic by afourney · Pull Request #6 · afourney/hamkit

afourney · 2026-10-19T03:30:24Z

Summary

APRS packet behaviour is now tested by one data-driven suite: JSONL case vectors in packages/aprs/tests/cases, checked by a single runner. The old corpus machinery and most packet-level Python tests are gone. Every packet row also records go-aprs-fap's output, and the differences are reviewed and explained in each row's comment. Running live APRS-IS traffic and FAP through the suite found several decoder and encoder problems, fixed here.

Case vectors

Format: each row has id, comment, tnc2, policy, decoded, reencoded, encoding_error, fap (documented in tests/cases/README.md). A row starts from received bytes (tnc2) or from a composed model (tnc2: null).
Rows: 5,335 across 19 family files. That's 3,919 received packets, each with a FAP snapshot, and 1,416 composed values. They come from:
- the original compliance corpus, migrated;
- roughly 1,800 rows converted from Python tests;
- traffic_sample.jsonl, a curated sample of a several-hour Pacific Northwest APRS-IS capture: the first two packets of each shape and software family.
The runner (tests/test_cases.py) checks every row every way it can:
- decoded JSON matches exactly (including -0.0 versus 0.0), and rebuilds the same value;
- serialize_packet, validate_packet, serialize_tnc2_line, check_packet and serialize_payload all agree;
- whatever is sent reads back cleanly and re-encodes to the same bytes;
- the value with received evidence removed is refused the same way or reads back with the same meaning.
tests/case_tool.py:
- propose [--write] shows rows whose expectations changed;
- fap fills in FAP snapshots;
- sweep CAPTURE [--fap BIN] [--add FILE] runs a raw capture through the parser, FAP and the runner, reporting failures, accept/reject differences from FAP, and new packet shapes.
tests/fap_snapshot/: a small Go harness pinned to go-aprs-fap commit 94c7118 (the same commit as the original FAP snapshots, which it reproduces exactly).
Comments: each comment records the row's provenance and reasoning, using searchable openings: Differs from FAP: …, Differs from FAP (representation only): …, Decision: … and Under this row's policy ….

Library changes

Weather: only 000/000 reads as unknown wind. A 000 direction with a non-zero speed is now sent as received; it used to be rewritten to 360.
Lenient decoding of usable but out-of-spec input. These now give a Packet with a nonconforming diagnostic instead of Malformed:
- an undecodable position timestamp, such as a time of day sent with z (the timestamp is left out);
- non-ASCII capability values;
- telemetry bits that aren't eight binary digits (kept as comment text);
- a Mic-E altitude containing non-base-91 bytes (kept as comment text).
Objects still require a valid timestamp. Malformed results now report a bad timestamp in their diagnostics. - Timestamps: an uppercase Z suffix on a position or object is read as z, with a
Compressed positions: PHG, DF signal, area and DF bearing text after a compressed position is now comment text (APRS101 p. 36: the compressed format "does not support PHG"). This matches what sending already refused.
/W and \W: treated as weather symbols only when weather fields follow (per aprs.org/aprs11/spec-wx.txt). A bare ddd/sss under W is now course and speed.
PARM/UNIT: trailing empty slots are no longer sent, so decoded definitions re-send as.

Removed

The old corpus and its machinery: tests/vectors/, generate_vectors.py, source.py, roundtrip_exceptions.json, fap_divergences.json,sequence_packet_expectations.json, the corpus runners and contract tests, and tests/fixtures/.
Python tests now covered by vectors or duplicated elsewhere, including all of tespy and tests/fields/test_codecs.py.
What's left in Python: tests of model construction, JSON contracts, parser options, documentation and computed properties, plus the fuzz test.

Testing

The full suite (7,825 tests) runs in about a minute, with ruff, mypy, black and prettier clean.
The case vectors pass on CPython 3.11, 3.12 and 3.15, and on PyPy 3.11.
sweep over the full 31,294-packet capture finds no runner failures and no packet shapes missing from the sample.

Known differences and open items

Where FAP is wrong, by APRS101, each case is recorded in the affected rows. The main ones:
- it ignores Mic-E position ambiguity;
- it reads 000/000 as zero instead of unknown;
- it reads area-object fields as motion;
- its Mic-E telemetry markers are reversed;
- it places 298 LoRa Mic-E packets with an invalid destination near the equator.
Judgment call: a Mic-E altitude at the end of the status text stays comment text; APRS101 doesn't say where it goes.
Not addressed:
- some Mic-E send errors are also reported at $.payload.device_code;
- serialize_payload on a Mic-E packet with an unknown message raises ConformanceEuiredError.

afourney · 2026-10-19T03:42:01Z

Review of head 1dca3395fcda391d3ce8b8fc3686c7f2dc8f633f against c0190a085ab5bd2524a354740d4bba64003535b3.

Recommendation: address the following before merging.

1. P2: Compressed area-like comment text is silently discarded

_extensions.py:124–146

Reproducer (Python bytes literal):

b'N0CALL>APRS:!\\5c!!<<!!l7@C102/503'

The new not compressed guard skips area decoding, but 102/503 then matches and is consumed by the course/speed branch. The position decoder overwrites that motion with compressed motion. The resulting comment is empty, no diagnostic is produced, and serialization drops the text. Under this PR's intended behavior, preserve it as comment text. The base retained an area extension and refused emission; silent loss is introduced here.

2. P2: North wind emits an out-of-spec direction

_weather.py:234–239

N0CALL>APRS:!4903.50N/07201.75W_000/010g020t070

This now serializes unchanged with 000/010. APRS101 p.27 specifies wind direction 001–360, with 000/000 the unavailable pair. Permissive reception should not make this valid emission: normalize north to 360/010, as the base did, or refuse it. Semantic comparison can accommodate 0°/360° equivalence.
APRS101

3. P2: W-symbol heuristic reinterprets incomplete weather as vehicle motion

_extensions.py:85–95

N0CALL>APRS:!4903.50N/07201.75WW180/010r001

This becomes motion (180, 10), no weather, and comment r001, with no diagnostics and successful emission. The base preserved wind/rain as weather, diagnosed missing required slots, and refused emission. The cited weather clarification makes W symbols interchangeable with weather symbols but does not establish this g/t-prefix requirement. Incomplete weather should not silently acquire a different meaning; any reinterpretation for nonstandard moving stations should be explicit.

4. P2: Replacement runner loses serializer refusal checks

test_cases.py:80–92

Rejected rows return before calling serialize_payload or serialize_tnc2_line. I verified that STA-030 still passes with both replaced by functions returning b"INVALID": neither is called. A successful status case also passes when serialize_payload incorrectly raises ContextRequiredError, because that exemption is accepted for every payload.

Restore refusal checks, distinguishing header-only failures from payload failures, and restrict context exemptions to models that actually need context.

5. P3: Proposal output hides signed-zero changes

case_tool.py:188–194

The outer comparison uses exact JSON comparison, while the per-field display uses Python equality. A change only in a coordinate's zero sign reports one differing row without printing the old/new decoded values. The same applies to 1 versus 1.0. Use the exact comparator when deciding which fields to display.

Validation and scope

All 1,564 original vector IDs and input packets survived migration.
Confirmed 5,335 rows, including 3,919 received packets with FAP snapshots.
Current suite: 7,824 passed with one distribution-metadata failure before installation; both packaging tests passed after editable installation.
Selected old tests against the new code: 776 passed, 2 failed, both involving W-symbol behavior.
Independent focused checks reproduced the defects and exercised timestamp recovery, uppercase-Z normalization, object timestamp requirements, telemetry-bit preservation/refusal, and unavailable versus measured-calm wind.
Tested on Python 3.12.14. The FAP harness could not be rebuilt because the available go command was not a Go compiler; the original full traffic capture was unavailable.

The exact snapshots and round trips are useful regression checks, but independently reviewed expectations remain important for protocol correctness.

afourney · 2026-10-19T04:01:35Z

Thanks for the review. All five points were valid and were introduced by this PR; they are fixed in f54a790.

1. Compressed area-like comment text silently discarded

After a compressed position no 7-byte data extension is read at all (course/speed, PHG, RNG, DFS, area, DF bearing): APRS101 pp. 27 and 36 place them only after uncompressed positions, and course/speed and range travel in csT. Text shaped like one stays comment and is sent unchanged, and the invalid-extension lookalike check no longer applies there. Your reproducer is now CMP-163: 102/503 is kept in the comment and re-sent.

2. North wind emitted as `000/010`

Reverted: north wind is always sent as 360 (APRS101 p. 27: 001–360, with 000/000 the unavailable pair). A received 000 direction with a non-zero speed decodes as north with a nonconforming diagnostic ("wind direction 000 is outside 001-360; north is sent as 360"). As you suggested, the runner's semantic comparison now treats 0° and 360° as the same direction for wind and course, which is why I had changed the encoder in the first place. WX-174 and WX-177 and their comments are updated.

3. W-symbol heuristic

Agreed that the g/t rule was not supported by spec-wx and changed meaning silently. /W and \W are again weather symbols by default, interchangeable with _; W180/010r001 is incomplete weather, diagnosed and refused (WX-182). Reading a W course/speed as station motion is now an explicit opt-in reinterpretive extension, w_symbol_motion, off by default, consistent with the other reinterpretive switches. Even when enabled it applies only when no weather field (any field letter, not just g/t) follows, so r001 keeps the report weather (WX-183). The mobile app in the capture decodes as wind by default (TRF-00288) and as motion with the extension (EXT-322). Documented in ENVELOPE_POLICIES.md, API_COVERAGE.md and JSON_FORMAT.md.

4. Runner refusal checks

For refused rows the runner now requires serialize_tnc2_line to refuse with exactly the same error, and serialize_payload to refuse with the same findings relative to the payload, or to succeed when every finding belongs to the header or envelope (including the "information exceeds N bytes" limit). The ContextRequiredError exemption is restricted to payloads that need header context (Mic-E, and NMEA carrying a symbol). Both of your demonstrations now fail: b"INVALID" stubs for both functions on STA-030, and ContextRequiredError from a status payload. One deliberate limit: for refused Mic-E rows the payload-alone result is not compared, because the destination is half the encoding and the packet and payload legitimately report different findings (e.g. MIC-672: an invalid destination at packet level, a missing speed for the payload alone).

5. `propose` hides signed-zero changes

The per-field display now uses the exact comparator; a received row differing only in the sign of a zero longitude now shows its old and new decoded.

Notes

Four new rows cover the reproducers (CMP-163, WX-182, WX-183, EXT-322), each with a FAP snapshot, and every comment that described the old W or wind behaviour was rewritten.
On independently reviewed expectations: agreed. FAP agreement is the independent check we have now; every FAP difference is explained in the row comments, and the runner could additionally assert that the fap snapshots are consistent if useful.
The packaging-test failure before installation and the FAP harness needing a Go 1.26 toolchain are environmental (tests/fap_snapshot builds with go build -o fapsnapshot .).

Full suite: 7,830 passed; ruff, mypy, black and prettier clean.

afourney

Re-review of f54a790: the original reproductions are fixed, but two adjacent P2 gaps remain. I recommend addressing these before merging.

Compressed W weather still becomes vehicle motion under the new opt-in policy.

With APRS_IS(extensions={"w_symbol_motion"}), parsing b"N0CALL>APRS:!/5c!!<<!!W7@Cr001" produces vehicle motion, weather=None, comment "r001", no diagnostics, and successful unchanged emission. The extension promises motion only when no weather field follows, but compressed detection recognizes only g, t, and c. Rainfall, humidity and pressure fields must also retain weather interpretation and trigger diagnostics/refusal for incomplete reports. Reproduced with both symbol tables.

Code:

hamkit/packages/aprs/src/hamkit/aprs/_extensions.py

Lines 111 to 115 in f54a790

    
           if symbol.code not in "_W" or w_symbol_motion(symbol, tail, policy): 
        
               return False 
        
           if not compressed and _COURSE_SPEED.match(tail): 
        
               return True 
        
           return tail.startswith((b"g", b"t", b"c"))

and

hamkit/packages/aprs/src/hamkit/aprs/_codec.py

Lines 347 to 352 in f54a790

    
           weather_symbol = symbol.code == "_" or ( 
        
               symbol.code == "W" 
        
               and ( 
        
                   "w_symbol_motion" not in policy.extensions 
        
                   or extensions.weather is not None 
        
               )

Rejected Mic-E payloads remain exempt from refusal assertions.

The runner invokes both serializers now, but skips checking the payload result for every Mic-E or symbol-bearing NMEA model. For MIC-424, serialize_payload correctly raises ConformanceError because altitude exceeds its wire range. Replacing it with a function returning b"INVALID" still lets test_case(row) pass. This altitude refusal is payload-local and does not depend on destination context. Restrict exemptions to actual context-dependent outcomes and assert payload-local refusals.

Code:

hamkit/packages/aprs/tests/test_cases.py

Lines 124 to 131 in f54a790

    
           relative = payload_relative(error) 
        
           # A Mic-E destination is half its encoding, so the payload alone is 
        
           # refused for what it can see; the packet check covers the rest. 
        
           if payload != "context" and not needs_context(value.payload): 
        
               if relative is None: 
        
                   assert isinstance(payload, bytes), payload 
        
               else: 
        
                   assert payload == relative

Resolved: compressed area-tail loss (including additional numeric/PHG/DFS/RNG comment checks), north-wind normalization to 360, default W-symbol classification and the original W repro, original status serializer mutations, and signed-zero proposal display.

Validation: full suite passed 7,830 tests plus 793 subtests on Python 3.12.14. Independent re-review had 31 passing checks and 11 failing checks: ten compressed-W variants and one refusal-coverage mutation. Six old W-symbol rejection tests now pass. FAP regeneration and the original full traffic capture remain unverified. Head was reverified at completion.

afourney · 2026-10-19T05:51:05Z

Thanks for the re-review. Both gaps were real; they are fixed in 7a6e1cb.

1. Compressed W weather read as motion

Confirmed, and the gap was wider than w_symbol_motion: after a compressed position only g, t and c started a weather report, so even under the default policy rain, humidity and pressure (r001, h50b09900) were left as comment text. Weather detection now recognises any weather field (a field letter from APRS101 pp. 62–66 or the APRS 1.1/1.2 additions, followed by a value character, so a comment word such as "home" is not taken as humidity). The same test is used after compressed positions, at the start of an uncompressed weather tail, and in the w_symbol_motion guard. Your reproducer !/5c!!<<!!W7@Cr001 is now incomplete weather, diagnosed and refused, under both symbol tables and both policies (WX-184..187); h50b09900 after a compressed _ decodes as weather (WX-188); a bare compressed W is motion only with the extension (EXT-323). No existing row changed.

2. Refused Mic-E payloads exempt

The runner no longer exempts Mic-E. For a refused Mic-E row it derives the expected serialize_payload result from the packet itself: the same packet with its destination left for the library to generate, encoded under a permissive envelope, with destination findings mapped to the payload root. The payload result must match that exactly. ContextRequiredError is now accepted only for NMEA carrying a symbol, the one payload that genuinely cannot be encoded without the header. Your b"INVALID" mutation now fails on all 88 refused Mic-E rows, MIC-424 included.

The stricter check exposed three library bugs, also fixed:

Mic-E status encoding errors without their own path were reported at $.payload.device_code. An out-of-range altitude is now reported at $.payload.extensions.altitude.measurement.value ("outside the Mic-E altitude range (-10000..743570 meters)"), and pole-cell and PHGR errors are no longer duplicated at device_code.
With an invalid destination, check_packet did not report a missing Mic-E speed; it now reports both (MIC-672).

25 Mic-E rows have updated expectations as a result.

Full suite: 7,836 passed plus 793 subtests; ruff, mypy, black and prettier clean.

afourney added 3 commits October 18, 2026 18:32

Move towards test vectors.

02ba7b4

Moved many tests to vectors.

97c8d6c

Removed unnecessary tests.

1dca339

Addressed reviewer comments.

f54a790

afourney commented Oct 19, 2026

View reviewed changes

Addressed reviewer comments.

7a6e1cb

afourney merged commit 950c143 into scaffold/aprs-module Oct 19, 2026
3 checks passed

Navigation Menu

Repository navigation

Move APRS tests to JSONL case vectors, compared against FAP and live traffic - #6

Move APRS tests to JSONL case vectors, compared against FAP and live traffic#6
afourney merged 5 commits into
scaffold/aprs-modulefrom
vectors

afourney commented Oct 19, 2026

Uh oh!

afourney commented Oct 19, 2026

Uh oh!

afourney commented Oct 19, 2026

Uh oh!

afourney left a comment

Uh oh!

afourney commented Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	if symbol.code not in "_W" or w_symbol_motion(symbol, tail, policy):
	return False
	if not compressed and _COURSE_SPEED.match(tail):
	return True
	return tail.startswith((b"g", b"t", b"c"))

	weather_symbol = symbol.code == "_" or (
	symbol.code == "W"
	and (
	"w_symbol_motion" not in policy.extensions
	or extensions.weather is not None
	)

	relative = payload_relative(error)
	# A Mic-E destination is half its encoding, so the payload alone is
	# refused for what it can see; the packet check covers the rest.
	if payload != "context" and not needs_context(value.payload):
	if relative is None:
	assert isinstance(payload, bytes), payload
	else:
	assert payload == relative

Conversation

afourney commented Oct 19, 2026

Summary

Case vectors

Library changes

Removed

Testing

Known differences and open items

Uh oh!

afourney commented Oct 19, 2026

1. P2: Compressed area-like comment text is silently discarded

2. P2: North wind emits an out-of-spec direction

3. P2: W-symbol heuristic reinterprets incomplete weather as vehicle motion

4. P2: Replacement runner loses serializer refusal checks

5. P3: Proposal output hides signed-zero changes

Validation and scope

Uh oh!

afourney commented Oct 19, 2026

1. Compressed area-like comment text silently discarded

2. North wind emitted as 000/010

3. W-symbol heuristic

4. Runner refusal checks

5. propose hides signed-zero changes

Notes

Uh oh!

afourney left a comment

Choose a reason for hiding this comment

Uh oh!

afourney commented Oct 19, 2026

1. Compressed W weather read as motion

2. Refused Mic-E payloads exempt

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

2. North wind emitted as `000/010`

5. `propose` hides signed-zero changes